1. Field of the Invention
The present invention relates generally to a rapid search system and method, and in particular to a rapid search system and method in high capacity hard disk drive clusters.
2. The Prior Arts
With the network transmission speed getting higher and cost lower, the digital video transmission over network becomes more popular, and meanwhile because the files of digital video are huge by nature, the size of digital storage medium is increasingly getting larger.
Since the size of the digital storage media is getting larger, a single hard disk, or even a single hard disk drive cabinet is not capable of meeting the mass storage requirement for a computer. As a result, two special communication protocols ATA over Ethernet (hereafter, refer to as AoE) and iSCSI are developed to allow the connection of mass storage systems over network.
AoE is a network communication protocol developed by Brantley Coile Company, designed for accessing ATA standard storage devices over ethernet network, therefore to achieve a storage area network through low cost standard technique.
AoE does not rely on network layers such as IP, UDP or TCP, etc. This also means that AoE is not routable through LAN, and therefore AoE is intended for storage area network only.
It is noted that both AoE and iSCSI all emphasize that they are the storage area network solution far cheaper than the Fibre Channel. However, AoE is simpler and cheaper to implement than iSCSI, and in regard to the technical document of the standard specification, the AoE specification is 8 pages compared with iSCSI's 257 pages.
With respect to the hardware support, currently, Coraid EtherDrive® is the hard disk drive cabinet designed for AoE. To the software support, vblade provided by SourceForge® allows the Linux Server installed with vblade to search for an AoE hard disk drive cabinet, and send request of storage service to the hard disk drive cabinet after having found it, whereby to use the storage space and resources thereof. Vblade has two types of status and implementations: a userspace one (part of aoetools package) and another one implemented as a linux kernel module.
Although AoE is a simple network protocol, it still increases the complexity of the storage mechanism, and the detail of AoE will be further described below.
AoE protocol mainly uses block storage transmission. In this case, ATA in AoE is regarded as a wire protocol in AoE. When data needs to be read after it is written to the hard disk, it is in one or more chunks of data called blocks, with each block being fixed-size.
AoE protocol directly and simply puts ATA command into lower-level network packets, which could use ethernet cable as an effective alternative to ribbon cable of ATA interface, and carry out the transmission of data blocks between each node of an ethernet network.
The ribbon cable of ATA standard doesn't care what the data blocks it is transferring, and neither does the AoE standard. Because ethernet cable used by AoE standard doesn't care what data block it is transferring, user could read and write any data block they need, but most of the time, file system is more suitable to organizing and structuring data than block.
Traditionally, the user uses a hard disk of a computer as a file system, and file systems such as ext3, XFS and NTFS are all designed and applied for such assumption. However, when using AoE, since AoE uses an ethernet cable to replace the ribbon cable of ATA, it is possible to violate this assumption. It is dangerous to run a traditional file system over AoE, which may lead to file system corruption or operating system kernel panic.
AoE uses the cluster file system to avoid these above-mentioned problems, which allows the block storage and transfer devices of AoE hard disks and hard disk drive cabinets to be shared by multiple computers accessing and using the devices, without the limitation of prior ATA technique that only allows one sole computer to access the hard disk and hard disk cluster. To the actual operation, the server for accessing an AoE hard disk drive cabinet is like a cluster computer part of a cluster system, which accesses the storage resources using the shared mechanism of the cluster system, and the AoE hard disk drive cabinet is a block storage device operable of sharing. GFS and OCFS2 etc. are commonly used cluster file systems.
With reference to FIG. 1, FIG. 1 is a schematic diagram showing a conventional storage area network. As shown in FIG. 1, in the conventional storage area network, the high capacity hard disk drive clusters mainly comprise a plurality of hard disk drive cabinets 10a˜10n, each hard disk drive cabinet comprises a plurality of hard disk drive drawers 11a, 11b, . . . , 11f, 11g, . . . , and each hard disk drive drawer comprises a plurality of hard disk drives 12a, 12b, . . . , 12f, 12g, . . . . When the computers 17a, 17b, . . . , 17g˜17n need to communicate with the hard disk drives 12a, 12b, . . . , 12f, 12g, . . . , AoE protocol or iSCSI protocol may be used.
In the AoE protocol, AoE packets use ethernet MAC address to indicate source and destination. A MAC address only works inside a single ethernet network because the MAC address is sent by broadcast in an ethernet broadcast domain. Current ethernet hardware features include flow control, which helps to minimize the bandwidth need for packet retransmission. Ethernet uses the cyclic redundancy checks (CRC) algorithm to ensure that packets arrive intact.
AoE and iSCSI protocols have flow control mechanism, which helps to minimize the bandwidth need for packet retransmission. However, it is still unavoidable that when hard disk drives 12a, 12b, . . . , 12f, 12g, . . . need to transmit data, for example when hard disk drives 12a and 12f have data to be transmitted among each other, they need to send broadcast packets to all the hard disk drives 12a, 12b, . . . , 12f, 12g, . . . , in order to find the designated hard disk drives 12a and 12f. If the number of the hard disk drives is not very large, these broadcast packets may not cause the breakdown of the whole storage system. However, once the number increases to a certain level, it is much possible to cause broadcast storm problem, leading to the breakdown of the whole storage system.